{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "71646c676fd2d912",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 【实验4-1】正则表达式  \n",
    "导入正则re库，并输入代码，输出结果，并解释前三行代码的意义。运行结果截图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "de62eb3fb5fe565a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-05T14:58:10.461449700Z",
     "start_time": "2024-05-05T14:58:10.447958400Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 41), match='Hello 123 4567 World_This is a Great Wall'>\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "content = 'Hello 123 4567 World_This is a Great Wall'\n",
    "result=re.match('^Hello\\s\\d\\d\\d\\s\\d{4}\\s\\w{10}\\s.*Wall$', content)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5e2a49712c2d48",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 【实验4-2】requests库"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3c19624a8dde0b",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "(1)安装requests库，并爬取百度首页。相关运行代码及运行结果截图。\n",
    "**其实按照[百度爬虫协议](https://www.baidu.com/robots.txt)的声明，[百度](https://www.baidu.com)除了部分`User-agent`可以爬取部分内容外，其余的`User-agent`全站都不允许意爬取，希望下次作业能换个可以合规爬取的网站**\n",
    "> 虽然爬虫协议有的人说是\"君子协议\"，我国法律并没有强制规定不得违反robots.txt,事实上网上每天都有数不清的不遵守robots.txt的爬虫在爬取数据。\n",
    "> \n",
    "> 但是事实上，像百度等大厂的法务部并不是吃干饭的，早已有不少相关案例。[百度诉360违反爬虫协议案宣判：360赔偿70万元](https://www.chinacourt.org/article/detail/2014/08/id/1361448.shtml)。\n",
    "> \n",
    "> 虽然我们学生怎么做，可能没什么，主要因为每天多的是爬虫在爬百度，我们算不了什么，而且人家就算故意碰瓷告了我们，也榨不出什么油水，根本就不屑于起诉我们。\n",
    "> \n",
    "> 但是如果今后毕业工作、或者实习时在公司这么干，稍不注意，对面法务就派人上门来了。人家就等着你大鱼上钩，给人家冲业绩。\n",
    "\n",
    "百度全站禁止爬取的说明如下（在[百度爬虫协议](https://www.baidu.com/robots.txt)文件结尾）\n",
    "```\n",
    "User-agent: *\n",
    "Disallow: /\n",
    "```\n",
    "运行以下powershell脚本 通过pip安装requests库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "271073c0a92005c8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-01T13:26:33.065583800Z",
     "start_time": "2024-05-01T13:26:31.303710400Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.mirrors.ustc.edu.cn/simple/\n",
      "Requirement already satisfied: requests in f:\\pythonproject\\py_web_programme\\regular_and_requests\\venv\\lib\\site-packages (2.31.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in f:\\pythonproject\\py_web_programme\\regular_and_requests\\venv\\lib\\site-packages (from requests) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in f:\\pythonproject\\py_web_programme\\regular_and_requests\\venv\\lib\\site-packages (from requests) (2.2.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in f:\\pythonproject\\py_web_programme\\regular_and_requests\\venv\\lib\\site-packages (from requests) (3.3.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in f:\\pythonproject\\py_web_programme\\regular_and_requests\\venv\\lib\\site-packages (from requests) (2024.2.2)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip available: 22.3.1 -> 24.0\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "!powershell pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5764734a29a0ae34",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "使用requests库爬取百度首页 得到百度网站页面 运行此代码块时 请关闭代理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2b6c85fa1cd4cab0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-05T14:58:56.054604600Z",
     "start_time": "2024-05-05T14:58:55.779416900Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\r\n",
      "<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class=\"bg s_ipt_wr\"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class=\"bg s_btn_wr\"><input type=submit id=su value=百度一下 class=\"bg s_btn\" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href=\"http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === \"\" ? \"?\" : \"&\")+ \"bdorz_come=1\")+ '\" name=\"tj_login\" class=\"lb\">登录</a>');\r\n",
      "                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style=\"display: block;\">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "response = requests.get('https://www.baidu.com', proxies={})\n",
    "response.encoding = response.apparent_encoding\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2717fd3182a5b40",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "\n",
    "(2)requests 库的六个主要方法是：\n",
    "\n",
    "    requests.get(url, params=None, **kwargs)：发送 GET 请求。\n",
    "    \n",
    "    requests.post(url, data=None, json=None, **kwargs)：发送 POST 请求。\n",
    "    \n",
    "    requests.put(url, data=None, **kwargs)：发送 PUT 请求。\n",
    "    \n",
    "    requests.delete(url, **kwargs)：发送 DELETE 请求。\n",
    "    \n",
    "    requests.head(url, **kwargs)：发送 HEAD 请求。\n",
    "    \n",
    "    requests.options(url, **kwargs)：发送 OPTIONS 请求。\n",
    "    \n",
    "(3)浏览器加载百度首页的过程大致如下：\n",
    "\n",
    "    浏览器向 DNS 服务器请求解析域名 www.baidu.com 的 IP 地址。\n",
    "    \n",
    "    浏览器向获取到的 IP 地址发送 HTTP 请求，请求百度首页的 HTML 文件。\n",
    "    \n",
    "    服务器接收到请求后，返回百度首页的 HTML 文件。\n",
    "    \n",
    "    浏览器接收到响应后，开始解析 HTML 文件，并发送额外的请求获取页面上的其他资源（例如 CSS、JavaScript、图片等）。\n",
    "    \n",
    "    浏览器根据接收到的资源渲染页面，呈现给用户。\n",
    "(4)Fiddler 的工作原理是:  \n",
    "\n",
    "    通过拦截计算机与服务器之间的通信流量来进行网络数据的调试和分析。它作为一个代理服务器，中间插入计算机和服务器之间的通信链路，可以捕获、查看、修改甚至篡改这些数据流量。 Fiddler 可以解密 HTTPS 流量，提供详细的请求和响应信息，以及方便的数据过滤、查找和修改功能，从而帮助开发者进行网络调试和性能优化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135bcc180b79830c",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 【实验4-3】参照教材P59实践项目：使用requests抓取黑马程序员论坛的数据。源代码级运行结果截图\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4df4293bc8ec437c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-05T15:20:55.389276700Z",
     "start_time": "2024-05-05T15:20:46.256501600Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在保存./crawled_files/黑马程序员论坛第1页.html\n",
      "爬取完第1页,并休息了0.4442165863983679秒\n",
      "正在保存./crawled_files/黑马程序员论坛第2页.html\n",
      "爬取完第2页,并休息了0.6287371557434898秒\n",
      "正在保存./crawled_files/黑马程序员论坛第3页.html\n",
      "爬取完第3页,并休息了0.5547465151546854秒\n",
      "正在保存./crawled_files/黑马程序员论坛第4页.html\n",
      "爬取完第4页,并休息了0.4321946688592635秒\n",
      "正在保存./crawled_files/黑马程序员论坛第5页.html\n",
      "爬取完第5页,并休息了0.6771948862708702秒\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import random\n",
    "#根据ur1发起请求，获取服务器响应文件ur1:待抓取的ur1\n",
    "def load_page(url):\n",
    "    #这里的请求头header可以是任意一个网站上面的请求头，进入开发者模式就可以找到，推荐使用google浏览器，比较方便查看#\n",
    "    header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0'}\n",
    "    #发送get请求\n",
    "    response =requests.get(url,headers=header)\n",
    "    response.encoding = \"utf-8\"\n",
    "    #返回html源代码\n",
    "    return response.text\n",
    "\n",
    "#将htm1文件保存为网页文件\n",
    "def save_file(html, filename):\n",
    "    print('正在保存'+filename) \n",
    "    with open(filename,'w',encoding='utf-8') as file:\n",
    "        file.write(html)\n",
    "#调度函数文\n",
    "def heima_fornum(begin_page, end_page):\n",
    "    for page in range(begin_page, end_page + 1):#组合页面完整的ur1\n",
    "        url=f'https://bbs.itheima.com/forum-231-{page}.html'\n",
    "        #上面的ur1以黑马程序员论坛的大数据技术交流模块为例子\n",
    "        file_name = './crawled_files/黑马程序员论坛第' + str(page) + '页.html'\n",
    "        # 抓取网页数据\n",
    "        html = load_page(url)\n",
    "        # 保存网页数据\n",
    "        save_file(html,file_name)\n",
    "        \n",
    "        # 随机休息一段时间 以免爬取速度过快 被识别为违法爬虫程序\n",
    "        delta_time = random.uniform(0.311451456,0.8114514)\n",
    "        time.sleep(delta_time)\n",
    "        print(f'爬取完第{page}页,并休息了{delta_time}秒')\n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    begin_page = int(input('请输入起始页:'))\n",
    "    end_page = int(input(\"请输出结束页:\"))\n",
    "    heima_fornum(begin_page, end_page)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99d2230f3e7dba52",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 【实验4-4】仿照教材P59实践项目，使用requests、正则表达式和文件操作爬取并保存“百度贴吧”某帖子全部内容。\n",
    "(已查询robots.txt，此部分内容未被禁止爬取，但是我试过频繁运行此代码会导致被百度识别为爬虫，然后爬到的内容全变成要验证码)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "579b72f4367f8fb7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-05T15:30:39.566583800Z",
     "start_time": "2024-05-05T15:30:31.400257900Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在爬取：https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn=50\n",
      "正在保存./crawled_files/百度贴吧第1页.html\n",
      "爬取完第1页,并休息了0.3183252499030906秒\n",
      "正在爬取：https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn=100\n",
      "正在保存./crawled_files/百度贴吧第2页.html\n",
      "爬取完第2页,并休息了0.5175482771500937秒\n",
      "正在爬取：https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn=150\n",
      "正在保存./crawled_files/百度贴吧第3页.html\n",
      "爬取完第3页,并休息了0.5859406208507378秒\n",
      "正在爬取：https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn=200\n",
      "正在保存./crawled_files/百度贴吧第4页.html\n",
      "爬取完第4页,并休息了0.6418979686008528秒\n",
      "正在爬取：https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn=250\n",
      "正在保存./crawled_files/百度贴吧第5页.html\n",
      "爬取完第5页,并休息了0.6444491651158661秒\n",
      "全部爬取完毕\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import random\n",
    "#根据ur1发起请求，获取服务器响应文件ur1:待抓取的ur1\n",
    "def load_page(url):\n",
    "    #这里的请求头header可以是任意一个网站上面的请求头，进入开发者模式就可以找到，推荐使用google浏览器，比较方便查看#\n",
    "    header = {\n",
    "        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0',\n",
    "        'Sec-Ch-Ua-Platform':\"Windows\"\n",
    "    }\n",
    "    #发送get请求\n",
    "    response =requests.get(url,headers=header)\n",
    "    response.encoding = response.apparent_encoding\n",
    "    #返回html源代码\n",
    "    return response.text\n",
    "\n",
    "#将htm1文件保存为网页文件\n",
    "def save_file(html, filename):\n",
    "    print('正在保存'+filename) \n",
    "    with open(filename,'w',encoding='utf-8') as file:\n",
    "        file.write(html)\n",
    "        \n",
    "def heima_fornum(begin_page, end_page):\n",
    "    for page in range(begin_page, end_page + 1):#组合页面完整的ur1\n",
    "        pn=int(page*50)\n",
    "        url=f'https://tieba.baidu.com/f?kw=%E5%AD%99%E7%AC%91%E5%B7%9D&ie=utf-8&pn={pn}'\n",
    "        print(\"正在爬取：\"+url)\n",
    "        file_name = './crawled_files/百度贴吧第' + str(page) + '页.html'\n",
    "        # 抓取网页数据\n",
    "        html = load_page(url)\n",
    "        # 保存网页数据\n",
    "        save_file(html,file_name)\n",
    "        \n",
    "        # 随机休息一段时间 以免爬取速度过快 被识别为违法爬虫程序\n",
    "        delta_time = random.uniform(0.311451456,0.8114514)\n",
    "        time.sleep(delta_time)\n",
    "        print(f'爬取完第{page}页,并休息了{delta_time}秒')\n",
    "    print(\"全部爬取完毕\")\n",
    "        \n",
    "if __name__ == '__main__':\n",
    "    begin_page = int(input('请输入起始页:'))\n",
    "    end_page = int(input(\"请输出结束页:\"))\n",
    "    heima_fornum(begin_page, end_page)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ae611033d938184",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
